System and method for gesture-based point of interest search

ABSTRACT

A user, such as the driver of a vehicle, to retrieve information related to a point of interest (POI) near the vehicle by pointing at the POI or performing some other gesture to identify the POI. Gesture recognition is performed on the gesture to generate a target region that includes the POI that the user identified. After generating the target region, information about the POI can be retrieved by querying a server-based POI service with the target region or by searching in a micromap that is stored locally. The retrieved POI information can then be provided to the user via a display and/or speaker in the vehicle. This process beneficially allows a user to rapidly identify and retrieve information about a POI near the vehicle without having to navigate a user interface by manipulating a touchscreen or physical buttons.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/367,526, filed Dec. 2, 2016, which itself is a continuation of U.S.application Ser. No. 14/333,369, filed Jul. 16, 2014, which itself is acontinuation of U.S. application Ser. No. 13/835,252, filed Mar. 15,2013 and issued as U.S. Pat. No. 8,818,716, both of which areincorporated by reference in their entireties. This application is alsorelated to U.S. application Ser. No. 13/228,395, entitled “Vehicle UserInterface System,” which is incorporated by reference herein in itsentirety.

BACKGROUND 1. Field of the Invention

The present invention relates generally to gesture recognition and inparticular to searching for a point of interest based on a gesture.

2. Description of the Related Arts

Vehicle technologies and features available to and controlled by adriver have advanced in recent years. For example, many vehicles featureintegrated computing systems with network connections that can be usedto retrieve and display a wide range of information. One key function ofvehicle-based computing systems is the ability to retrieve informationrelated to points of interest (POI) near the vehicle. This can beuseful, for example, when the driver wishes to identify a nearbybuilding or view information (e.g., ratings and reviews) for arestaurant or store.

A driver typically interacts with a vehicle-based computing system byinputting commands via a touchscreen or physical buttons on the centerconsole of the vehicle. However, using a touchscreen or buttons torequest POI information by navigating a map or typing in a search termcan be cumbersome and frustrating, especially when the driver isrequesting information about a POI that he can see through the vehicle'swindows.

SUMMARY

A computing system retrieves information associated with a point ofinterest based on an identifying gesture that a user performs inside avehicle. The identifying gesture is oriented so that it identifies anobject outside the vehicle. The computing system receives a data signalrepresenting the identifying gesture and performs gesture recognition onthe data signal to determine a direction vector corresponding to thedirection of the identifying gesture. The system also accesses locationdata to identify the vehicle's current location and orientation. Thedirection vector, location, and orientation are then analyzed togenerate a target region that corresponds to the object that wasidentified by the gesture, and the system retrieves informationassociated with one or more POIs in the target region. The retrievedinformation is provided to the user via an output device, such as aspeaker or display.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the embodiments of the present invention can be readilyunderstood by considering the following detailed description inconjunction with the accompanying drawings.

FIG. 1 illustrates an exemplary operating environment 100 for variousembodiments of the gesture-based POI search system.

FIG. 2 is a high-level block diagram illustrating components of the POIinformation retrieval module of FIG. 1, according to one embodiment.

FIG. 3 is a flow chart illustrating a process for retrieving informationabout a POI based on an identifying gesture, according to oneembodiment.

FIG. 4A-4D illustrate an example of a gesture-based POI search.

FIG. 5 is a flow chart illustrating a process for maintaining micromapsin the gesture-based POI search system, according to one embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments are now described with reference to the accompanyingfigures. Like reference numbers indicate identical or functionallysimilar elements. Also in the figures, the left most digit of eachreference number corresponds to the figure in which the reference numberis first used.

Overview

A POI information retrieval module allows a user, such as the driver ofa vehicle, to retrieve information related to a point of interest nearthe vehicle by pointing at the POI or performing some other gesture toidentify the POI. A camera system in the vehicle captures the gestureand sends a data signal representing the gesture to the POI informationretrieval module. The POI information retrieval module performs gesturerecognition on the data signal to generate a target region that includesthe POI that the user identified. After generating the target region,information about the POI can be retrieved by querying a server-basedPOI service with the target region or by searching in a micromap that isstored locally. The retrieved POI information can then be provided tothe user via a display and/or speaker in the vehicle. This processbeneficially allows a user to rapidly identify and retrieve informationabout a POI near the vehicle without having to navigate a user interfaceby manipulating a touchscreen or physical buttons.

The user may optionally issue a voice command along with a gesture. If amicrophone in the vehicle detects a voice command, the POI informationretrieval module performs voice recognition on the voice command togenerate a character string representing the words that were spoken aspart of the command. The character string can then be used to helpidentify the POI that the user pointed at. For example, if the user says“building” while pointing at a building, the POI information retrievalmodule can ignore information for non-building objects (e.g.,playgrounds, parking lots, etc.) when retrieving information for POIs inthe target region.

Operating Environment

FIG. 1 illustrates an exemplary operating environment 100 for variousembodiments. The operating environment 100 may include an in-vehiclecommunications system 112. One example of such a system is an in-vehiclehands free telephone (HFT) controller 113 which will be used as anexample herein for ease of discussion. The operating environment 100 mayalso include a wireless mobile communication device (MCD) 102, acommunication link 105 for communications between the in-vehicle system112 and a network 120, a short-range communication link 109 forcommunication between the in-vehicle system 112 and the wireless mobilecommunication device 102, a wireless networking communication link 107between the wireless mobile communication device 102 and the network120, and a POI data server 122 connected to the network 120. Thecommunication links described herein can directly or indirectly connectthese devices. The network 120 can be a wireless communication networksuch as a cellular network comprised of multiple base stations,controllers, and a core network that typically includes multipleswitching entities and gateways, for example.

The functions described herein are set forth as being performed by adevice in the operating environment 100 (e.g., the in-vehiclecommunication system 112, the MCD 102, and/or the remote server 122). Inembodiments, these functions can be performed in any of these devices orin any combination of these devices and/or other devices.

The operating environment 100 includes input devices, such as a camerasystem 132, location sensors 133, and a microphone 134. The camerasystem 132, location sensors 133, and/or microphone 134 can be part ofthe in-vehicle system 112 (as shown in FIG. 1) or can be in the MCD 102(not shown), for example. In one embodiment, the camera system 132includes a sensor that captures physical signals from within the vehicle(e.g., a time of flight camera, an infrared sensor, a traditionalcamera, etc). The camera system 132 is positioned to capture physicalsignals from a user such as hand or arm gestures from a driver orpassenger. The camera system 132 can include multiple cameras positionedto capture physical signals from various positions in the vehicle, e.g.,driver seat, front passenger seat, second row seats, etc. Alternatively,the camera system 132 may be a single camera which is focused on oneposition (e.g., the driver), has a wide field of view, and can receivesignals from more than one occupant of the vehicle, or can change itsfield of view to receive signals from different occupant positions.

In another embodiment, the camera system 132 is part of the MCD 102(e.g., a camera incorporated into a smart phone), and the MCD 102 can bepositioned so that the camera system 132 captures gestures performed bythe occupant. For example, the camera system 132 can be mounted so thatit faces the driver and can capture gestures by the driver. The camerasystem 132 may be positioned in the cabin or pointing toward the cabinand can be mounted on the ceiling, headrest, dashboard or otherlocations in/on the in-vehicle system 112 or MCD 102.

After capturing a physical signal, the camera system 132 outputs a datasignal representing the physical signal. The format of the data signalmay vary based on the type sensor(s) that were used to capture thephysical signals. For example, if a traditional camera sensor was usedto capture a visual representation of the physical signal, then the datasignal may be an image or a sequence of images (e.g., a video). Inembodiments where a different type of sensor is used, the data signalmay be a more abstract or higher-level representation of the physicalsignal.

The location sensors 133 are physical sensors and communication devicesthat output data associated with the current location and orientation ofthe vehicle. For example, the location sensors 133 may include a devicethat receives signals from a global navigation satellite system (GNSS)or an electronic compass (e.g., a teslameter) that measures theorientation of the vehicle relative to the four cardinal directions. Thelocation sensors 133 may also operate in conjunction with thecommunication unit 116 to receive location data associated withconnected nodes in a cellular tower or wireless network. In anotherembodiment, some or all of the location sensors 133 may be incorporatedinto the MCD 102 instead of the vehicle.

The microphone 134 captures audio signals from inside the vehicle. Inone embodiment, the microphone 134 can be positioned so that it is moresensitive to sound emanating from a particular position (e.g., theposition of the driver) than other positions (e.g., other occupants).The microphone 134 can be a standard microphone that is incorporatedinto the vehicle, or it can be a microphone incorporated into the MCD102. The microphone 134 can be mounted so that it captures voice signalsfrom the driver. For example, the microphone 138 may be positioned inthe cabin or pointing toward the cabin and can be mounted on theceiling, headrest, dashboard or other locations in/on the vehicle or MCD102.

The POI information retrieval module 136 retrieves information relatedto one or more POIs based on input from the camera system 132 and(optionally) the microphone 134. After performing the search, the module136 sends the result to the display 138 and/or speaker 140 so that theresult can be provided to the user. A detailed description of thecomponents and operation of the POI information retrieval module 136 ispresented below with reference to FIGS. 2-5.

The operating environment 100 also includes output devices, such as adisplay 138 and a speaker 140. The display 138 receives and displays avideo signal. The display 138 may be incorporated into the vehicle(e.g., an LCD screen in the central console, a HUD on the windshield),or it may be part of the MCD 102 (e.g., a touchscreen on a smartphone).The speaker 140 receives and plays back an audio signal. Similar to thedisplay 138, the speaker 140 may be incorporated into the vehicle, or itcan be a speaker incorporated into the MCD 102.

The in-vehicle hands-free telephone (HFT) controller 113 and wirelessmobile communication device (MCD) 102 may communicate with each othervia a short-range communication link 109 which uses short-rangecommunication technology, such as, for example, Bluetooth® technology orother short-range communication technology, for example, UniversalSerial Bus (USB). The HFT controller 113 and mobile communicationsdevice 102 may connect, or pair, with each other via short-rangecommunication link 109. In an embodiment the vehicle 113 can include acommunications unit 116 that interacts with the HFT controller 113 toengage in the short range communications, a memory unit device 114, anda processor 118. The HFT controller 113 can be part of a vehicle'stelematics system which includes memory/storage, processor(s) andcommunication unit(s). The HFT controller 113 can utilize the vehicle'stelematics unit to assist in performing various functions. For example,the communications unit 116 and/or processor 118 can be part of thevehicle's telematics unit or can be a separate unit in the vehicle.

The processors 108, 118 and/or 128 process data signals and may comprisevarious computing architectures including a complex instruction setcomputer (CISC) architecture, a reduced instruction set computer (RISC)architecture, or an architecture implementing a combination ofinstruction sets. Although only a single processor is shown in eachdevice in FIG. 1, multiple processors may be included in each device.The processors can comprise an arithmetic logic unit, a microprocessor,a general purpose computer, or some other information appliance equippedto transmit, receive and process electronic data signals from the memory104, 114, 124, and other devices both shown and not shown in thefigures.

Examples of a wireless mobile communication device (MCD) 102 include acellular phone, personal device assistant (PDA), smart phone, pocketpersonal computer (PC), laptop computer, tablet computer, smart watch orother devices having a processor, communications capability and areeasily transportable, for example. The MCD 102 includes a communicationsunit 106, a memory unit device 104, and a processor 108. The MCD 102also includes an operating system and can include various applicationseither integrated into the operating system or stored in memory/storage104 and executed by the processor 108. In a common form, an MCDapplication can be part of a larger suite of vehicle features andinteractions. Examples of applications include applications availablefor the iPhone™ that is commercially available from Apple Computer,Cupertino, Calif., applications for phones running the Android™operating system that is commercially available from Google, Inc.,Mountain View, Calif., applications for BlackBerry devices, availablefrom Research In Motion Ltd., Waterloo, Ontario, Canada, and/orapplications available for Windows Mobile devices, available fromMicrosoft Corp., Redmond, Wash.

In alternate embodiments, the mobile communication device 102 can beused in conjunction with a communication device embedded in the vehicle,such as a vehicle-embedded phone, a wireless network card, or otherdevice (e.g., a Wi-Fi capable device). For ease of discussion, thedescription herein describes the operation of the embodiments withrespect to an embodiment using a mobile communication device 102.However, this is not intended to limit the scope of the embodiments andit is envisioned that other embodiments operate using othercommunication systems between the in-vehicle system 112 and the network120, examples of which are described herein.

The mobile communication device 102 and the in-vehicle system 112 mayexchange information via short-range communication link 109. The mobilecommunication device 102 may store information received from thein-vehicle system 112, and/or may provide the information (such as voiceand/or gesture signals) to a remote processing device, such as, forexample, the remote server 122, via the network 120. The remote server122 can include a communications unit 126 to connect to the network 120,for example, a memory/storage unit 124 and a processor 128.

In some embodiments, the in-vehicle system 112 may provide informationto the mobile communication device 102. The mobile communication device102 may use that information to obtain additional information from thenetwork 120 and/or the server 122. The additional information may alsobe obtained in response to providing information with respect to aprompt on wireless mobile communication device 102 from in-vehiclesystem 112.

The network 120 may include a wireless communication network, forexample, a cellular telephony network, as well as one or more othernetworks, such as, the Internet, a public-switched telephone network(PSTN), a packet-switching network, a frame-relay network, a fiber-opticnetwork, and/or other types of networks.

Performing Gesture-Based Point of Interest Searches

FIG. 2 is a high-level block diagram illustrating components of the POIinformation retrieval module 136 of FIG. 1, according to one embodiment.The POI information retrieval module 136 includes a gesture recognitionmodule 202, a location module 204, a voice recognition module 206, andinput analysis module 208, a POI search module 210, a micromapmanagement module 212, micromap storage 214, and an information outputmodule 216. In alternative embodiments, the POI information retrievalmodule 136 may include additional, fewer, or different components, andthe functionality of the components 202 through 216 described herein maybe distributed among components of the information retrieval module 136in a different manner.

The gesture recognition module 202 receives a data signal from thecamera system 132 and performs a gesture recognition algorithm on thereceived data signal to identify and interpret the gesture that wascaptured by the camera system 132. As described above with reference tothe camera system 132, the data signal is an electronic representationof a gesture that the user performed in the vehicle. For example, thedata signal may be an image of the gesture, a sequence of images, orsome other representation of the gesture. In one embodiment, the gesturerecognition module 202 is configured to automatically detect anidentifying gesture that identifies an object exterior to the vehicle.When an identifying gesture is detected, the gesture recognition module202 analyzes the gesture to determine a direction vector representingthe direction of the gesture.

The voice recognition module 204 receives an output signal from themicrophone 134 and performs a voice recognition algorithm on thereceived signal to identify voice commands received by the microphone134. The voice recognition module 204 generates a computer-readableoutput representing words in the voice command. For example, the voicerecognition module 204 may output the words as a character string.

The location module 206 receives data from the location sensors 133 anduses the data to determine the current location and orientation of thevehicle. If the location module 206 receives multiple types of data fordetermining the vehicle's current location (e.g., a combination of GNSSdata and location data for connected cell towers), then the module 206may perform averaging or some other aggregation technique to combine thedata into a single location (e.g., a single set of lat/longcoordinates). The location module 206 may similarly perform aggregationto combine multiple types of orientation data.

The input analysis module 208 receives input data from the gesturerecognition module 202, the voice recognition module 204, and thelocation module 206 and analyzes the input data to determine a targetregion corresponding to an identifying gesture that was captured by thecamera system 132. After determining the target region, the inputanalysis module 208 queries the POI search module 210 and/or themicromap search module 211 to retrieve information related to points ofinterest inside the target region. The operation of the input analysismodule 208 is described below in greater detail.

The POI search module 210 receives a target region from the inputanalysis module 208 and performs a point of interest search in thetarget region by querying a remote server. In addition to the targetregion, the POI search module 210 may also receive character stringsrepresenting voice commands issued by the user. In this case, the POIsearch module 210 may include the character strings in the query inorder to obtain more accurate results. To perform the search, the POIsearch module 210 may access a database on the server 122.Alternatively, the module 210 may access a service operating on athird-party server (e.g., Yelp™, Google Local).

The micromap search module 211 receives a target region from the inputanalysis module 208 and searches a corresponding micromap in themicromap storage 214 for information related to POIs in the targetregion. As used herein, a micromap is a map of a region that containsone or more POIs. Each micromap also includes information related to thePOIs in the micromapped region. Since micromaps are stored locally inthe micromap storage 214 in some embodiments, POI information that isstored in a micromap can be accessed with less latency than POIinformation that is retrieved from a remote server (e.g., by the POIsearch module 210). This can be particularly beneficial in regions witha high density of POIs, such as a downtown region in a major city.

The micromap management module 212 retrieves micromaps of regions thatthe vehicle is likely to enter and stores the retrieved micromaps in themicromap storage. In one embodiment, the micromap management module 212monitors the location, orientation, and speed of the vehicle toautomatically identify micromaps for retrieval. An example process forautomatically identifying micromaps in this manner is described indetail with reference to FIG. 5.

The data output module 216 receives information related to one or morePOIs from one of the search modules 210, 211 and sends the informationto the display 138, the speaker 140, or some other output device in theMCD 102 or the in-vehicle communications system 112. In one embodiment,the data output module 216 sends an audio representation of a portion ofthe information for a point of interest while showing additionalinformation to the user via the display 138. For example, if the dataoutput module 216 receives information related to a restaurant, themodule 216 may have the speaker 140 speak out the name of the restaurantwhile reviews of the restaurant are sent to the display 138.

In other embodiments, some or all of the components 202 through 216 ofthe POI information retrieval module 136 are positioned external to thein-vehicle system 112. In one embodiment, the components 202 through 216are implemented as an application downloaded to the MCD 102 (e.g.,applications available from iTunes). In another embodiment, thecomponents 202 through 216 are implemented on the remote server 122, anddata from the camera system 132, location sensors 133, and microphone134 are sent over the network 120 to the remote server 122 to beanalyzed.

FIG. 3 is a flow chart illustrating a process for retrieving informationabout a POI based on an identifying gesture, according to oneembodiment. For ease of discussion, the process 300 shown in FIG. 3 willbe described below in conjunction with the example shown in FIGS. 4A-4D.The process 300 begins when a user performs an identifying gestureinside a vehicle. The identifying gesture is directed toward theexterior to the vehicle to identify an object outside the vehicle andrequest information about the object. For example, the user shown inFIG. 4A is performing an identifying gesture to request information fora building near the vehicle 402 by pointing at the building with anoutstretched arm and forefinger. As the user performs the identifyinggesture, the camera system 132 captures 302 the identifying gesture andsends a data signal representing the gesture to the gesture recognitionmodule 202.

After receiving the data signal from the camera system 132, the gesturerecognition module 202 performs 304 gesture recognition on the datasignal to determine a direction vector corresponding to the identifyinggesture. In one embodiment, the gesture recognition module 202 usesdepth information in the data signal to generate a 3D depthreconstruction of the identifying gesture. The 3D depth reconstructionis then used to determine the direction vector 404. An example directionvector 404 is illustrated in FIG. 4B. In the illustrated example, thedirection vector 404 is a two-dimensional vector that represents thedirection of the identifying gesture relative to a pair of axes 406 thatare parallel and perpendicular to the vehicle's direction of travel.

In addition to the outstretched arm and forefinger gesture shown in theexample of FIG. 4A, the gesture recognition module 202 may also beconfigured to recognize other types of identifying gestures anddetermine a direction vector upon detecting one of these other gestures.For example, the gesture recognition module 202 may also be configuredto determine a direction vector for an identifying gesture comprising anoutstretched arm without a specific arrangement of fingers, or anidentifying gesture comprising a hand on the steering wheel with anoutstretched finger pointing at an exterior object.

The user may optionally issue a voice command to provide additionalinformation about the object being identified. In the exampleillustrated in FIG. 4A, the user says “building” while performing theidentifying gesture to indicate that he is pointing at a building. Avoice command may be particularly helpful in situations where the objectbeing identified (e.g., a building) is adjacent to a different type ofobject (e.g., a park). Although the user shown in FIG. 4A issues thevoice command at the same time as he performs identifying gesture, theuser may alternatively issue a voice command before or after performingthe identifying gesture.

If the microphone 134 captures a voice command with the identifyinggesture, then the voice recognition module 204 analyzes the voicecommand to generate a computer-readable representation of the command.In the example of FIG. 4A, the voice recognition module 204 generatesthe character string “building” after receiving the corresponding audiosignal from the microphone 134.

Meanwhile, the location module 206 receives data from the locationsensors 133 to determine 306 the current location and orientation of thevehicle. In one embodiment, the location module 206 polls the locationsensors 133 to determine a current location and orientation only afterdetecting that the user has performed an identifying gesture. In anotherembodiment, the location module 206 polls the location sensors 133 atregular intervals to maintain a constantly updated location andorientation for the vehicle.

Next, the input analysis module 208 receives the direction vector andthe current location and orientation of the vehicle and generates 308 atarget region that is likely to contain the object that the useridentified. In one embodiment, the target region is generated to alignwith the direction vector. In the example shown in FIG. 4B, the targetregion 408 has an elongated triangular shape that follows the directionof the direction vector 404 and has one corner anchored at the locationof vehicle 402. The triangular shape shown in the example of FIG. 4B isa particularly convenient shape because it can be defined as a geo-fencewith three pairs of lat/long coordinates (i.e., defining the vertices ofthe triangle). Alternatively, the target region may be some other shapethat corresponds to the direction vector. In one embodiment, the inputanalysis module 208 also uses the current speed of the vehicle whendetermining the target region. For example, the target region can extendfarther from the vehicle when the vehicle is traveling at a fasterspeed.

After generating 308 the target region, the input analysis module 208accesses the micromap storage 214 to determine 310 whether the targetregion overlaps with any micromaps that have been stored in the micromapstorage 214. If the target region does not overlap with any micromappedregions, then the input analysis module 208 sends the target region tothe POI search module 210, and the POI search module 210 performs asearch to retrieve 312 information for the POI that was identified withthe identifying gesture. As described above with reference to FIG. 2,the POI search module 210 may access the remote server 122 or athird-party service (e.g., Yelp™, Google Local) to perform a POI searchin the target region. The POI information may include, for example, aname for the POI, a short description, images, hours of operation,contract information, ratings and reviews, and other information.

If a voice command was received with the gesture, then the inputanalysis module 208 also passes a character string representing thevoice command to the POI search module 210 so that the character stringcan be used to narrow the results of the POI search. For example, thePOI search module 210 would perform a search for the term “building”within the target region 408 after receiving the inputs shown in FIG.4A.

Since the user typically performs the identifying gesture with theintention of retrieving information about a single POI, the inputanalysis module 208 may use an iterative process to adjust the targetregion until the POI search module 210 returns a single POI. Forexample, a triangular target region (e.g., the example target region 408shown in FIG. 4B) can be adjusted by changing the length of the trianglein the direction of the direction vector 404 and/or changing the angleof the triangle at the corner corresponding to the vehicle 402. Thus, ifthe POI search finds multiple POIs, the input analysis module 208 mayiteratively decrease the size of the target region until one POI isfound. Similarly, if the search does not return any POIs, the inputanalysis module 208 may iteratively increase the size of the targetregion until a POI is returned. The single POI is then sent to the dataoutput module 216 to be provided to the user. Alternatively, the inputanalysis module 208 may merely use the iterative process to reduce thenumber of POIs but still send multiple POIs to the data output module208. This may be useful in cases where there is uncertainty over whichPOI the user was attempting to identify. For example, if there are twobuildings in close proximity to each other in the example target region408 of FIG. 4B, then the input analysis module 208 may send both POIs tothe data output module 216.

If the input analysis module 208 determines that the target regionoverlaps with a micromapped region, then the input analysis module 208sends the target region to the micromap management module 212 so thatthe micromap search module 211 can search the corresponding micromap toretrieve 314 POI information for the identified POI. The input analysismodule 208 and the micromap search module 211 may operate in conjunctionto perform an iterative process similar to the process described abovewith reference to the POI search module 210 to narrow the POIinformation that is sent to the data output module 216. Since themicromap is stored locally, the iterative process can be performed morequickly. In addition, the micromap search module 211 can advantageouslyperform a search in a locally stored micromap in situations where thecommunication links 105, 107 to the network 120 are unreliable orunavailable. Micromaps are also advantageous because they provideincreased granularity in identifying and localizing POIs, and such POIsmay reference various types of establishments. With localization,micromapping also enables more accurate reconstruction. In oneembodiment, the range of reconstruction is limited to the range ofmicromapped objects. Hence, using a micromap may also change the rangeand overall number of accessible POIs. In one embodiment, the inputanalysis module 208 retrieves POI information from both search modules210, 211 in parallel and merges the two sets of retrieved POIinformation into a single set of results. For example, the inputanalysis module 208 uses an artificial intelligence unit to merge theretrieved POI information.

After receiving the POI information from the input analysis module 208,the data output module 216 provides 316 the POI information to the userusing the various output devices in the vehicle or the MCD 102. The dataoutput module 216 may be configured to use one output device to output aportion of the POI information and use a different output device tooutput additional information. For example, the name of a POI may bespoken out to the user using the speaker 140 (shown in FIG. 4C) whilethe display 138 is used to show more detailed information about the POI,such as a description, photos, and contact information (shown in FIG.4D).

In the example shown in FIGS. 4C-4D, the data output module 216 ismerely outputting information for a single POI. However, if the inputanalysis module 208 send information for multiple POIs to the dataoutput module 216, the data output module 216 may first output a list ofPOIs (e.g., by showing a visual interface on the display 138 or by usingthe speaker 140 to speak out the names of the POIs). The user can thenselect a POI from the list (e.g., by performing a pointing gesture atthe display 138 or speaking out a voice command) to view additionalinformation for the POI.

In another embodiment, the gesture recognition module 202 determines 304a three-dimensional direction vector instead of a two-dimensionalvector, and the rest of the process 300 is expanded into threedimensions. Thus, location module 206 also determines 306 the vehicle'saltitude, and the input analysis module 208 generates 308 athree-dimensional target region (e.g., a cone). Using athree-dimensional process 300 can beneficially provide more accurate POIinformation in locations where multiple POIs have the same lat/longcoordinates but are located a different altitudes. For example, supposethe vehicle is driving through a city. If the user points toward the topof a skyscraper, the POI information retrieval module 136 would provideinformation about the skyscraper's observation deck instead ofinformation about restaurants in the skyscraper's lobby.

FIG. 5 is a flow chart illustrating a process 500 for maintainingmicromaps in the gesture-based POI search system, according to oneembodiment. The micromap management module 212 begins by obtaining 502the current speed, orientation, and location of the vehicle. Thelocation and orientation of the vehicle may be obtained from thelocation module 206, as described above with reference to FIG. 2. In oneembodiment, the micromap management module 212 obtains the vehicle'sspeed by accessing a component of the vehicle that directly measures itsspeed (e.g., a speedometer). Alternatively, the micromap managementmodule 212 may determine the vehicle's speed by analyzing the locationdata of the vehicle over a known period of time.

Next, the micromap management module 212 analyzes the speed,orientation, and location data of the vehicle to identify 504 upcomingmicromapped regions that the vehicle is likely to pass by or travelthrough. In one embodiment, the module 212 generates a retrieval regionin front of the vehicle based on the vehicle's speed, orientation, andlocation, and any micromapped regions inside the retrieval region areidentified as upcoming micromapped regions. The retrieval region mayhave a triangular shape similar to the target region described abovewith reference to FIGS. 3 and 4B. Alternatively, the retrieval regionmay have some other shape (e.g., a corridor centered on the road thatthe vehicle is currently traveling on, an ellipse that extends in frontof the vehicle, etc).

After identifying 504 one or more upcoming micromapped regions, themicromap management module retrieves 506 the corresponding micromapsfrom the remote server 122. As described above with reference to themicromap management module 212, a micromap is a map of a region thatcontains one or more POIs and contains information related to the POIsin the micromapped region. The POI information in a micromap mayinclude, for example, a name for the POI, a short description, images,hours of operation, contract information, ratings and reviews,performance schedules, and other information. The retrieved micromaps506 are stored 508 in the micromap storage 214 so that they can berapidly accessed when the user performs an identifying gesture for a POIin one of the stored micromaps.

In addition to automatically adding micromaps according to the process500 described above, the micromap management module 212 may also beconfigured to delete micromaps from the micromap storage 214. In oneembodiment, the module 212 deletes micromaps based on a similar analysisof the vehicle's speed, orientation, and location. For example, themicromap management module 212 may automatically delete a micromap ifthe vehicle is moving away from the corresponding region. In anotherembodiment, the module 212 may delete a micromap if the micromap has notbeen accessed for a certain period of time.

Additional Considerations

Although the description herein is presented with reference to anin-vehicle communications system 112, the systems and processesdescribed in this specification may also be implemented in mobiledevices such as smartphones and tablet computers independently of avehicle. For example, a magnetometer and location module integrated intoa mobile device can be used to determine the location, speed, andorientation of a mobile device, while a camera in the mobile device canbe used to capture the identifying gesture.

Reference in the specification to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least oneembodiment. The appearances of the phrase “in one embodiment” or “anembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Some portions of the detailed description that follows are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps (instructions)leading to a desired result. The steps are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical, magnetic or opticalsignals capable of being stored, transferred, combined, compared andotherwise manipulated. It is convenient at times, principally forreasons of common usage, to refer to these signals as bits, values,elements, symbols, characters, terms, numbers, or the like. Furthermore,it is also convenient at times, to refer to certain arrangements ofsteps requiring physical manipulations or transformation of physicalquantities or representations of physical quantities as modules or codedevices, without loss of generality.

However, all of these and similar terms are to be associated with theappropriate physical quantities and are merely convenient labels appliedto these quantities. Unless specifically stated otherwise as apparentfrom the following discussion, it is appreciated that throughout thedescription, discussions utilizing terms such as “processing” or“computing” or “calculating” or “determining” or “displaying” or“determining” or the like, refer to the action and processes of acomputer system, or similar electronic computing device (such as aspecific computing machine), that manipulates and transforms datarepresented as physical (electronic) quantities within the computersystem memories or registers or other such information storage,transmission or display devices.

Certain aspects of the embodiments include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the embodiments couldbe embodied in software, firmware or hardware, and when embodied insoftware, could be downloaded to reside on and be operated fromdifferent platforms used by a variety of operating systems. Theembodiments can also be in a computer program product which can beexecuted on a computing system.

The embodiments also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for thepurposes, e.g., a specific computer, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, such as, but is notlimited to, any type of disk including floppy disks, optical disks,CD-ROMs, magnetic-optical disks, read-only memories (ROMs), randomaccess memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards,application specific integrated circuits (ASICs), or any type of mediasuitable for storing electronic instructions, and each coupled to acomputer system bus. The memory/storage can be transitory ornon-transitory. Memory can include any of the above and/or other devicesthat can store information/data/programs. Furthermore, the computersreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may also be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the method steps. The structure for a variety ofthese systems will appear from the description below. In addition, theembodiments are not described with reference to any particularprogramming language. It will be appreciated that a variety ofprogramming languages may be used to implement the teachings of theembodiments as described herein, and any references below to specificlanguages are provided for disclosure of enablement and best mode.

In addition, the language used in the specification has been principallyselected for readability and instructional purposes, and may not havebeen selected to delineate or circumscribe the inventive subject matter.Accordingly, the disclosure of the embodiments is intended to beillustrative, but not limiting, of the scope of the embodiments, whichare set forth in the claims.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative methods and systems for performing agesture-based POI search. Thus, while particular embodiments andapplications of the present invention have been illustrated anddescribed, it is to be understood that the invention is not limited tothe precise construction and components disclosed herein and thatvarious modifications, changes and variations which will be apparent tothose skilled in the art may be made in the arrangement, operation anddetails of the method and apparatus of the present invention disclosedherein without departing from the spirit and scope of the invention asdefined in the appended claims.

What is claimed is:
 1. A vehicle-based computer-implemented method forretrieving information associated with a point of interest, the methodcomprising: accessing, at a computer system, electronic movement datarepresenting a current speed of a vehicle and electronic location datarepresenting a current location and an orientation of the vehicle basedon an electronic reception of electronic sensor data; analyzing theelectronic movement data and the electronic location data to determine aregion that the vehicle is likely to travel through; receiving anelectronic data signal representing a gesture performed by a user insidethe vehicle; performing gesture recognition on the electronic datasignal to determine a direction vector representing a direction thegesture indicates, wherein the direction vector is a multi-dimensionalvector that represents a direction of the gesture performed by the userrelative to a pair of axes that are parallel to a direction of travel ofthe vehicle and perpendicular to the direction of travel of the vehicle;receiving a voice command issued by the user inside the vehicle;performing electronic voice recognition on the voice command toelectronically determine one or more words spoken by the user as part ofthe voice command; analyzing, at the computer system, the directionvector, the electronic movement data, the electronic location data, theregion that the vehicle is likely to travel through and the words spokenby the user to identify a point of interest (POI) that is of interest tothe user that aligns with the direction vector, wherein a point ofinterest search is performed by electronically accessing andelectronically querying an electronic datastore to determine informationassociated with the POI; and executing electronic instructions toelectronically control a display device to provide the informationassociated with the POI.
 2. The computer-implemented method of claim 1,further including generating a target region corresponding to thedirection vector, the electronic movement data, and the electroniclocation data; wherein identifying the POI includes determining that thePOI is inside the target region, wherein the target region aligns withthe direction vector and is dependent on the current speed of thevehicle.
 3. The computer-implemented method of claim 2, wherein a sizeof the target region increases as a speed of the vehicle increases. 4.The computer-implemented method of claim 2, wherein: depth informationwithin the data signal is utilized to generate a three-dimensional depthreconstruction of the gesture that pertains to the target region,wherein the location data includes an altitude of the vehicle, and thetarget region is three-dimensional based on the three-dimensional depthreconstruction of the gesture.
 5. The computer-implemented method ofclaim 2, further including: accessing from a local storage, informationassociated with one or more points of interest that are located in theregion that the vehicle is likely to travel through, wherein the one ormore points of interest include the POI; and determining locations ofthe one or more points of interest relative to the target region;wherein the POI is identified from among the one or more points ofinterest by determining that the POI is inside the target region.
 6. Thecomputer-implemented method of claim 5, wherein if the informationassociated with the POI is not in the local storage, then the methodfurther includes: querying a remote server with the target region toidentify the POI, and accessing the information associated with the POIfrom the remote server.
 7. The computer-implemented method of claim 6,wherein if two or more points of interest, including the POI, are insidethe target region and are identified as being of interest to the user,then the method further includes using the one or more words spoken bythe user to identify the POI from among the two or more points ofinterest.
 8. The computer-implemented method of claim 6, wherein if twoor more points of interest, including the POI, are inside the targetregion and are identified as being of interest to the user, then themethod further includes iteratively decreasing a size of the targetregion until only the POI is inside the target region.
 9. Thecomputer-implemented method of claim 6, wherein: if two or more pointsof interest, including the POI, are inside the target region and areidentified as being of interest to the user, then the two or more pointsof interest, including the POI, is provided to the user in a list; andthe information associated with the POI is provided to the user upon thePOI being selected from the list by the user.
 10. Thecomputer-implemented method of claim 6, wherein if zero points ofinterest are inside the target region, then the method further includesiteratively increasing a size of the target region until only the POI isinside the target region.
 11. A non-transitory computer-readable storagemedium for storing computer program instructions for retrievinginformation associated with a point of interest (POI), the programinstructions when executed by a processor cause the processor to performsteps including: accessing electronic movement data representing acurrent speed of a vehicle and electronic location data representing acurrent location and an orientation of the vehicle based on anelectronic reception of electronic sensor data; analyzing the electronicmovement data and the electronic location data to determine a regionthat the vehicle is likely to travel through; receiving an electronicdata signal representing a gesture performed by a user inside thevehicle; performing gesture recognition on the electronic data signal todetermine a direction vector representing a direction the gestureindicates, wherein the direction vector is a multi-dimensional vectorthat represents a direction of the gesture performed by the userrelative to a pair of axes that are parallel to a direction of travel ofthe vehicle and perpendicular to the direction of travel of the vehicle;receiving a voice command issued by the user inside the vehicle;performing electronic voice recognition on the voice command toelectronically determine one or more words spoken by the user as part ofthe voice command; analyzing, at the computer system, the directionvector, the electronic movement data, the electronic location data, theregion that the vehicle is likely to travel through and the words spokenby the user to identify a POI that is of interest to the user thataligns with the direction vector, wherein a point of interest search isperformed by electronically accessing and electronically querying anelectronic datastore to determine information associated with the POI;and executing electronic instructions to electronically control adisplay device to provide the information associated with the POI. 12.The non-transitory computer readable storage medium of claim 11, whereinthe program instructions further cause the processor to perform stepsincluding generating a target region corresponding to the directionvector, the electronic movement data, and the electronic location data;wherein identifying the POI includes determining that the POI is insidethe target region, wherein the target region aligns with the directionvector and is dependent on the current speed of the vehicle.
 13. Thenon-transitory computer readable storage medium of claim 12, wherein asize of the target region increases as a speed of the vehicle increases.14. The non-transitory computer readable storage medium of claim 12,wherein: depth information within the data signal is utilized togenerate a three-dimensional depth reconstruction of the gesture thatpertains to the target region, wherein the location data includes analtitude of the vehicle, and the target region is three-dimensionalbased on the three-dimensional depth reconstruction of the gesture. 15.The non-transitory computer readable storage medium of claim 12, whereinthe program instructions further cause the processor to perform stepsincluding: accessing from a local storage, information associated withone or more points of interest that are located in the region that thevehicle is likely to travel through, wherein the one or more points ofinterest include the POI; and determining locations of the one or morepoints of interest relative to the target region; wherein the POI isidentified from among the one or more points of interest by determiningthat the POI is inside the target region.
 16. The non-transitorycomputer readable storage medium of claim 15, wherein if the informationassociated with the POI is not in the local storage, then the programinstructions further cause the processor to perform steps including:querying a remote server with the target region to identify the POI, andaccessing the information associated with the POI from the remoteserver.
 17. The non-transitory computer readable storage medium of claim16, wherein if two or more points of interest, including the POI, areinside the target region and are identified as being of interest to theuser, then the program instructions further cause the processor toperform steps including using the one or more words spoken by the userto identify the POI from among the two or more points of interest. 18.The non-transitory computer readable storage medium of claim 16, whereinif two or more points of interest, including the POI, are inside thetarget region and are identified as being of interest to the user, thenthe program instructions further cause the processor to perform stepsincluding iteratively decreasing a size of the target region until onlythe POI is inside the target region.
 19. The non-transitory computerreadable storage medium of claim 16, wherein: if two or more points ofinterest, including the POI, are inside the target region and areidentified as being of interest to the user, then the two or more pointsof interest, including the POI, is provided to the user in a list; andthe information associated with the POI is provided to the user upon thePOI being selected from the list by the user.
 20. The non-transitorycomputer readable storage medium of claim 16, wherein if zero points ofinterest are inside the target region, then the program instructionsfurther cause the processor to perform steps including iterativelyincreasing a size of the target region until only the POI is inside thetarget region.